Multi-Instance Dimensionality Reduction

Multi-instance learning deals with problems that treat bags of instances as training examples. In single-instance learning problems, dimensionality reduction is an essential step for high-dimensional data analysis and has been studied for years. The curse of dimensionality also exists in multiinstance learning tasks, yet this difficult task has not been studied before. Direct application of existing single-instance dimensionality reduction objectives to multi-instance learning tasks may not work well since it ignores the characteristic of multi-instance learning that the labels of bags are known while the labels of instances are unknown. In this paper, we propose an effective model and develop an efficient algorithm to solve the multi-instance dimensionality reduction problem. We formulate the objective as an optimization problem by considering orthonormality and sparsity constraints in the projection matrix for dimensionality reduction, and then solve it by the gradient descent along the tangent space of the orthonormal matrices. We also propose an approximation for improving the efficiency. Experimental results validate the effectiveness of the proposed method. Introduction In single-instance scenario we are given a training set containing N instances with their labels. In multi-instance learning (Dietterich, Lathrop, and Lozano-Perez 1997) the training examples are N bags each containing many instances. The labels of training bags are known yet the labels of the training instances are unknown. According to the standard multi-instance learning assumption, a positive bag contains at least one positive instance, while all the instances in negative bags are negative. Multi-instance learning has been found useful in modeling many real world applications such as drug activity prediction (Dietterich, Lathrop, and Lozano-Perez 1997), image retrieval (Andrews, Tsochantaridis, and Hofmann 2003), text categorization (Andrews, Tsochantaridis, and Hofmann 2003), face detection (Viola, Platt, and Zhang 2006), computer-aidedmedical diagnosis (Fung et al. 2007), ∗Supported by the National Fundamental Research Program of China (2010CB327903), the National Science Foundation of China (60635030, 60721002) and Jiangsu Science Foundation (BK2008018). Copyright c © 2010, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. etc. Many of these tasks involve high-dimensional data and thus encounter the curse of dimensionality. In single-instance scenario the curse of dimensionality has attracted much attention. There are two major paradigms, i.e., feature selection and dimensionality reduction. Feature selection tries to select a subset of the original features according to some measurements such as the mutual information or distance-based measures. Raykar et al. (2008) have studied multi-instance feature selection using Bayesian method which automatically considers the feature relevance. In most cases, searching an optimal feature subset is hard and heuristic methods are often used. Dimensionality reduction, which tries to extract a small number of new features by projecting the original features into a new space, is generally with better theoretical foundation. Existing dimensionality reduction techniques can be roughly divided into two categories, that is, unsupervised approaches such as PCA (principal component analysis) (Jolliffe 2002), and supervised approaches such as LDA (linear discriminant analysis) (Fukunaga 1990). To the best of our knowledge, multiinstance dimensionality reduction has not been studied before. It is noteworthy that multi-instance dimensionality reduction is even harder than single-instance dimensionality reduction since the input space of multi-instance learning task is ambiguous. In this paper, we propose the MIDR (Multi-Instance Dimensionality Reduction) approach based on a specifically designed dimensionality reduction objective for multiinstance learning. We formulate the objective as an optimization problem by considering orthonormality and sparsity constraints in the projection matrix for dimensionality reduction, and then solve it by gradient descent along the tangent space of the orthonormal matrices. We also propose an approximation to improve the efficiency. Experimental results validate the effectiveness of the proposed method. The rest of this paper is organized as follows. We start by a brief review of related work. Then, we propose MIDR and report our experiments, which is followed by the conclusion.